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Improving Decision Making Skill Using an Online Volcanic Crisis 
Simulation: Impact of Data Presentation Format 

Elizabeth J. Barclay , 1 Carl E. Renshaw, 1,a) Holly A. Taylor , 2 and A. Reyan Bilge 2 

ABSTRACT 

Creating effective computer-based learning exercises requires an understanding of optimal user interface designs for 
improving higher order cognitive skills. Using an online volcanic crisis simulation previously shown to improve decision 
making skill, we find that a user interface using a graphical presentation of the volcano monitoring data reduces the effec¬ 
tiveness of the exercise compared to an informationally equivalent text-based data presentation format. Results are con¬ 
sistent with earlier work demonstrating that the over-automation of quantitative analyses in computer-based learning 
reduces their effectiveness in improving higher order skills. Additional research is critically needed to clarify the condi¬ 
tions under which user interfaces can be optimized for ease of use while not sacrificing the exercises' effectiveness in 
improving higher order cognitive skills. ©2011 National Association of Geoscience Teachers. [DOI: 10.5408/1.3543933] 


INTRODUCTION 

Designing effective computer aided instruction (CAI) 
exercises (i.e., those that improve critical thinking skills) 
requires the delineation of specific instructional methodolo¬ 
gies and user interfaces that positively impact higher order 
cognition. Much of the existing emphasis on computer user 
interface design is on increasing the "usability" of software 
— a concept that emphasizes the ease of learning, efficiency 
of use, as well as user satisfaction (Butler, 1996; Green and 
Payne, 1984; Hombaek, 2006; Nielsen, 1994; Rubens et al., 
2005). An important question that arises, however, is to 
what extent minimizing learning effort and speeding up 
performance is compatible with the objectives of educa¬ 
tional CAI software that is meant to engage the user and to 
effectively convey the desired concepts (Ormrod, 1995). For 
example, in previous work (Renshaw et al., 2000) we have 
shown that while automating quantitative analyses in a CAI 
exercise reduces completion time, it also reduces its educa¬ 
tional effectiveness; the exercise more effectively improves 
student mastery of the complex task of dimensional analysis 
when they perform the underlying algebraic analyses them¬ 
selves rather than having the computer do the algebraic 
analyses automatically. These results indicate that students 
need to fully understand the underlying analyses before the 
core conceptual ideas become clear, an idea known as 
"scaffolding," referring to the idea that students build new 
knowledge upon existing knowledge (Brown et al., 1989). 
However, the appropriate level of automation is population 
dependent. Automating basic algebraic tasks for students 
proficient in these skills may decrease learning time without 
a corresponding loss of effectiveness (Renshaw et al, 1998). 

The idea that less automation leads to greater learning 
is consistent with previous work showing that user interfa¬ 
ces optimized for ease of use and minimal cognitive effort 
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are not always effective for learning purposes (Svendsen, 
1991; Trudel and Payne, 1995). These studies suggest that 
the more mental effort exerted in understanding a concept, 
the better it is learned (Ormrod, 1995; Salomon, 1979). This 
conclusion is consistent with the cognitive theory of gener¬ 
ative learning and supporting studies that find greater 
learning is achieved when knowledge is actively generated 
using, at least in part, existing knowledge (e.g., Slamecka 
and Graf, 1978; Wills et al, 2000; Wittrock, 1989). 

However, the conclusion that greater mental effort 
results in better learning does not necessarily hold with 
respect to the graphical representation of data. For exam¬ 
ple, we have shown that students trained using CAI to 
interpret three-dimensional data using color-coded maps 
have greater understanding of the underlying data than 
those trained to use traditional monochrome contour maps 
requiring greater mental effort to interpret (Taylor et al, 
2004). These results suggest that effective graphical presen¬ 
tation of data may facilitate its interpretation while 
decreasing cognitive effort. 

From the CAI user interface design perspective, the au¬ 
tomation of algebraic tasks and the presentation of data in 
graphical form are similar in that both reduce the time 
required to complete the exercise and hence increase soft¬ 
ware usability. However, from a cognitive perspective, alge¬ 
braic manipulations and the interpretation of graphical data 
require different cognitive resources. Thus, it is not surpris¬ 
ing that their incorporation into CAI software differentially 
impacts an exercise's effectiveness in improving higher 
order cognition. In one case, increasing usability by auto¬ 
mating algebraic analyses reduced student learning, but in 
another case increasing usability by enhancing the graphical 
representation of data increased student understanding. 
These results highlight a critical uncertainty in the optimal 
CAI software design: it is essential that we understand how 
different approaches to enhancing usability impact the soft¬ 
ware's effectiveness in improving higher order cognition. 

Here we investigate the optimal user interface design for 
presenting scalar data in an online CAI interactive simulation 
of a volcanic hazard designed to improve decision-making 
skill. As appears to be the case for three-dimensional (e.g., 
spatial) data, it may be that presenting scalar data graphically 
enhances student understanding of the data and thus 
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improves the exercise's impact on higher order skills. Alter¬ 
natively, the greater cognitive effort required to interpret sca¬ 
lar data presented in a simple text-based format may 
promote a fuller understanding of the data, leading to better 
learning and greater impact on higher order cognition. 

The online CAI simulation used here previously has 
been shown to improve decision making skill (Taylor et al., 
1997). Below we give an overview of the simulation and 
review the methodology used to assess its impact on stu¬ 
dent decision-making skill. We then describe how the data 
presentation format in the simulation was modified. The 
impact of the different data presentation formats in the 
simulation on student decision-making skill was then 
assessed using the same methodology used previously and 
the results compared to determine the impact of data pre¬ 
sentation format on the CAI simulation's effectiveness in 
improving decision making. 

BACKGROUND 

Eruption 

In the CAI laboratory Eruption, students take on the 
role of a volcanologist who must purchase equipment, 
interpret scalar data (instrument readings), and decide 
whether a village should be evacuated (Fig. 1). Eruption 
was originally developed by volcanologists at the State 
University of New York at Buffalo (Sheridan et al., 1993). 
As part of the work presented here, the Eruption code was 
updated and ported to a web-based format now freely 
available at http://www.dartmouth.edu/~earthsci/erup- 
tion. The fundamental design objective of the simulation is 
to require students to use scientific data in decision mak¬ 
ing. In the role of a volcanologist, students obtain informa¬ 
tion from three instruments commonly used to monitor 
volcanoes: a geodimeter, a correlation spectrometer (cospec), 


and a seismometer. The geodimeter provides information 
about ground deformation; the cospec about gas emissions; 
and the seismometer about earthquake frequency. In the 
simulation, as in reality, information from these instruments 
is imperfectly correlated with volcanic eruptions, making 
instrument readings related to different probability estimates 
of eruption. The correlations between instrument readings 
and eruption probability can be framed either positively or 
negatively based on how the outcome is described. For 
example, a cospec reading of 400 tons S02/day can be associ¬ 
ated with a 60% chance of eruption (a negative framing 
because an eruption is a negative outcome) or a 40% chance 
that the volcano remains dormant (a positive framing 
because continued dormancy is a positive outcome). 

Prior to using Eruption, students determine the correla¬ 
tion between instrument readings and volcanic activity by 
analyzing a previous period of volcanic unrest. The vol¬ 
canic activity in the simulation itself and that presented for 
the previous period of unrest is synthesized based on 
documented sequences from various volcanoes around the 
world. Students see the synthesized volcanic activity for 
the previous period of volcanic unrest summarized in a ta¬ 
ble comparing instruments readings to observations of vol¬ 
canic activity taken every 12 h for a 25 day period. 
Volcanic activity is consistently framed either positively 
(e.g., the volcano did or did not remain "dormant") or neg¬ 
atively (e.g., the volcano did or did not "erupt"). From 
these data, students develop a correlation table for each 
instrument, with each table classifying the specific instru¬ 
ment's readings into three ranges representing low, inter¬ 
mediate, and high readings (e.g., cospec readings of <400 
tons S0 2 /day, 400-750 tons/day, and >750 tons/day, 
respectively). For each range of readings, students calculate 
the probability, based on the historical record, that the vol¬ 
cano remains dormant over the next 12 h if the data are 
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FIGURE 1: Text-based version of the Eruption simulation. 


















































































J. Geosci. Educ. 59, 85-92 (2011) 


Improving Decision Making Skill 87 


positively framed, or the probability that the volcano erupts 
over the next 12 h if the data are negatively framed. We 
refer to the framing of the correlation tables as the "Eruption 
framing," i.e., the framing of the exercise, to distinguish it 
from the "volcanic hazard scenario framing," described 
below, that is used to measure decision-making skill. 

Assessing Decision Making Skill 

The educational objectives of Eruption are to serve as a ve¬ 
hicle for introducing various techniques used to monitor vol¬ 
canic hazards and to improve student decision-making skill. 
Here we focus on the later objective and begin by reviewing 
the methodology for assessing decision-making skill. 

When making decisions, people grant differential im¬ 
portance to available information. Various factors, includ¬ 
ing how the information is presented, can influence this 
weighting. A phenomenon known as the "framing effect" 
illustrates the influence of context on decisions (e.g., Tver- 
sky and Kahneman, 1982). For example, framing a medical 
decision either in terms of survival or mortality affects the 
choice between two procedures (McNeil et al., 1982), even 
when the alternate frame can be determined. Thus the like¬ 
lihood of an individual agreeing to undergo a risky opera¬ 
tion differs depending on whether they are told that they 
have a 95% chance of surviving the operation (positive 
framing) versus a 5% chance that they will die (negative 
framing) (Tversky and Kahneman, 1982). The inconsis¬ 
tency in decision making due to framing is a robust effect 
observed across a variety of contexts ranging from medi¬ 
cine (McNeil et al., 1982) to law (McCaffery et al., 1995) to 
finance (Tversky and Kahneman, 1986) to environmental 
hazards (Taylor et al., 1997) and across different popula¬ 
tions including different ages and levels of education (Tay¬ 
lor et al., 1997; Tversky and Kahneman, 1986) and 
nationalities (McNeil et al., 1982). Only when the alternate 
framework is specifically presented does the discrepancy 
in decision behavior disappear (Jou et al., 1996). Inconsis¬ 
tent and poor decisions arise in part because individuals 
fail to consider alternate frameworks. 

Since the consideration of alternate frameworks is an 
essential component of good decision making, we argue 
that a person's sensitivity to framing is one measure of 
their decision-making skill. We emphasize that this strat¬ 
egy for assessing decision making skill does not require the 
definition of a "correct" decision, only the assertion that a 
"good" decision is one that is consistent and independent 
of the way data have been framed. 

In previous work we documented inconsistent deci¬ 
sions due to problem framing in a variety of environmental 
problems across a spectrum of populations ranging from 
high school students to geoscience professionals (Taylor 
et al., 1997). Of particular interest here are the framing 
effects associated with a volcanic hazard problem based on 
the political policy scenario used by Quattrone and Tver¬ 
sky (1988). We refer to this problem as the "volcanic haz¬ 
ard scenario." The positively framed version of this 
scenario is: 

Batuna, a small nation on a volcanic island, wishes to 
improve their ability to predict volcanic eruptions. The 
nation has one hundred million dollars ($100M) to 
invest in two prediction systems. A decision on how 
the $100M will be distributed between two systems 


must be made. Both systems have been shown to be 
equally accurate at predicting eruptions. 

Field studies indicate that the first prediction system is 
in working order 91% of the time whereas the second 
technology is in working order 97% of the time. 

How much of the $100M would you spend for each 
system? The total amount spent on both program 
should add up to $100M. 

The negatively framed version of this scenario is 
identical except for the second paragraph, which is 
presented as 

Field studies indicate that the first prediction system 
randomly breaks down 9% of the time, while the 
second prediction system breaks down 3% of the time. 

Taylor et al. (1997) showed that undergraduates en¬ 
rolled in introductory geoscience courses at the State Uni¬ 
versity of New York at Buffalo and Tufts University 
allocated, on average, twice as much money to the first sys¬ 
tem when the question was positively framed compared to 
when the question was negatively framed; the allocation of 
funds between the two systems was nearly equal in the 
positive framing, but 3:1 in favor of the more reliable sys¬ 
tem in the negative framing. Although demographic data 
were not collected, the students in these classes were 
drawn from across the institutions and likely are represen¬ 
tative of the overall demographics of these schools: nearly 
all of the participants were likely within a few years of 20 
years old, predominantly Caucasian, and more or less 
equally distributed between genders. As noted above, 
however, framing effects are robust across demographics 
and context. For example, Quattrone and Tversky (1988), 
using a structurally similar problem addressing crime rates 
given to undergraduates at Standard University and the 
University of California at Berkeley, also found that the 
allocation students made between the two alternatives was 
more even when the question was positively framed. They 
ascribe this framing effect to what they term the "ratio-dif¬ 
ference principle" which states that the perceived impact 
of any fixed absolute difference between two amounts 
increases with their ratio. Thus the difference between 3% 
and 9% (ratio: 9/3 = 3) is perceived as greater than the dif¬ 
ference between 91% and 97% (ratio: 97/91 ~1), even 
though the absolute difference (6%) is the same in both 
cases. Thus the allocation of funds between the two sys¬ 
tems is more even in the positive framing (ratio ~1) 
because there is less perceived difference between the two 
systems than in the negative framing (ratio = 3). 

To assess the impact of Eruption on student decision¬ 
making skill, as measured by their sensitivity to framing 
effects, Taylor et al. (1997) gave a questionnaire including the 
volcanic hazard scenario to 270 high school students from 
western New York after they completed the Eruption exercise. 
Half of the students used the computer-based version of 
Eruption, half used an informationally equivalent paper-based 
version. Responses to the volcanic hazard scenario among 
high school students who used the paper-based version of 
Eruption were similar to those of high school students who 
had never used either version of Eruption in that the 
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FIGURE 2: Graphic-based version of the Eruption simulation. 


responses of both groups showed a significant framing effect 
[paper-based users: F(1, 269) = 14.033, MSe=351.028, 
p < 0.001]. In contrast, no significant sensitivity to framing of 
the volcanic hazard scenario was present among those who 
had used the computer-based Eruption [computer-based 
users: F(l, 147) = 0.30, MSe = 1241.95, p = 0.58]. Unprompted 
comments among students using the computer-based Erup¬ 
tion demonstrate their explicit recognition of the alternative 
framings during their decision making: many noted that an 
X% probability of surviving was equal to a (100-X)% proba¬ 
bility of dying. Comments of this ilk were not observed with 
students using the paper-based version. 

Modifications to Eruption 

The central task of students using the Eruption exercise 
is to obtain information from three instruments used to 
monitor volcanoes and, based on this information, decide 
to what extent a community threatened by the volcano 
should be evacuated. Our objective is to determine how 
the presentation format of the scalar instrument data 
impacts the exercise's effectiveness in improving decision¬ 
making skill. 

Accordingly, we explore two different approaches for 
conveying the instrument readings. In the text-based pre¬ 
sentation format version, the current instrument readings 
appear as numerical values (Fig. 1). The student must com¬ 
pare these values to each instrument's correlation table to 
determine the likelihood of imminent volcanic activity. In 
the graphics-based presentation format version, the instru¬ 
ment data appear on a color-coded graph depicting the 
instrument reading over time (Fig. 2). Shading on the 
graph divides the instrument readings into three ranges, 
low, intermediate, and high, corresponding to the correla¬ 
tion table ranges, making it readily apparent to which 
range each data point belongs. Data labels for both presen¬ 


tation formats are identical, e.g., in both cases seismicity is 
quantified in units of events per day, etc. Aside from the 
data presentation format, the two versions of Eruption are 
identical. 


METHODS 

To assess the relative impact of the two versions of 
Eruption on decision making, 129 undergraduates from 
Dartmouth College completed the online, web-based Erup¬ 
tion exercise and a postexercise questionnaire including the 
volcanic hazard scenario. The students completed the exer¬ 
cise and questionnaire either as part of a regularly sched¬ 
uled laboratory associated with an introductory geoscience 
course or as an optional out-of-class assignment as part of 
an introductory course on natural hazards. Demographic 
data were not collected, but as the courses draw students 
seeking to fulfill their science distribution requirement 
from across the institution, the demographics of the partici¬ 
pants are likely similar to those of the institution as a 
whole: nearly all of the participants were likely within a 
few years of 20 years old, predominantly Caucasian, and 
more or less equally distributed between genders. 

The study used a 2 ( Eruption frame: positive, 
negative) x 2 (data format: text, graphical) x 2 (volcanic 
hazard scenario frame: positive, negative) between-partici- 
pant design. Students were randomly assigned to each 
condition, either based on lab section (introductory geosci¬ 
ence courses) or individually (optional out-of-class assign¬ 
ment). In all cases, prior to using Eruption students 
received a short lecture introducing the equipment used to 
monitor volcanic hazards and an explanation of why data 
from these instruments are related to the volcanic activity. 
The lecture also included an introduction to Eruption and a 
demonstration of its use. 
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TABLE I: Means, standard deviations (in parentheses) for amount of 
money (in millions of dollars) allocated to program A as a function of 
Eruption interface. Eruption frame, and volcanic hazard scenario frame. 
Numbers of students in each condition also provided. 



Graphics-based Interface 

Text-based Interface 

Eruption frame 

Positive 

Negative 

Positive 

Negative 

Positive hazard Frame 

27.3 (15.5) 

34.5 (22.6) 

33.7 (16.3) 

25.2 (14.2) 

N = 10 

N = 15 

N = 24 

N = 16 

Negative hazard frame 

36.1 (20.9) 

22.5 (16.9) 

29.1 (18.7) 

26.8 (12.8) 

N = 14 

N = 15 

N = 20 

N = 15 


RESULTS 

The analysis consisted of a univariate analysis of var¬ 
iance (ANOVA) using the dollar amount the student allo¬ 
cated to the first prediction system as the dependent 
variable (see Table I). The data met the assumptions for 
using a parametric ANOVA. Analyses revealed a signifi¬ 
cant interaction between Eruption frame, data presentation 
format, and volcanic hazard scenario frame [F(l, 
121) = 4.52, MSe = 1387.51, p<0.05]. Follow-up simple 
effects (Kirk, 1982; Maxwell, 1990; Winer et al., 1991) 
explored the locus of this interaction by examining interac¬ 
tions between Eruption frame and volcanic hazard frame 
separately for the two data formats. These follow-up analy¬ 
ses revealed no significant interactions for students who 
used the text-based format [F(l, 121) = 0.566, MSe = 307.31, 
p > 0.05] but among the students who used the graphics- 
based format a significant interaction remained between 
Eruption frame and volcanic hazard frame [F(l, 
121) = 4.622, MSe = 307.31, p<0.01; see Fig. 3], Thus the 
decision making of those students who used the graphics- 
based version of Eruption is less skilled in the volcanic haz¬ 
ard scenario in the sense that their decisions are influenced 
by the framing of the volcanic hazard scenario whereas the 
decision making in the volcanic hazard scenario of stu¬ 
dents who used the text-based presentation format is 
invariant to the framing of the scenario. This is shown in 
Fig. 3 which compares the funds allocated to the first (less 
reliable) system in the volcanic hazard scenario as a func¬ 
tion of scenario frame. Among students who used the 
graphics-based version of Eruption [Fig. 3(a)], the funds 
allocated to the first system is sensitive to the framing of 
both Eruption and the volcanic hazard scenario. In contrast, 
the allocation of funds to the first system among students 


who used the text-based version of Eruption is not sensitive 
to either the framing of Eruption or the hazard scenario. 

DISCUSSION 

Factors Impacting Volcanic Hazard Scenario Decisions 

The significant interaction between data presentation 
format, Eruption frame, and volcanic hazard scenario frame 
demonstrates that data presentation format impacts Erup¬ 
tion's success in improving decision making. Indeed, only 
students who used the text-based data presentation format 
subsequently made decisions in the hazard scenario invari¬ 
ant to information framing, either of the scenario or Erup¬ 
tion itself. As stated previously, this invariance to 
information format results in better information-based 
decisions. 

In contrast to the responses of students who used the 
text-based presentation format, responses to the volcanic 
hazard scenario among students who used the graphics- 
based format varied as a function of Eruption framing. 
Among this group, participants who used the negatively 
framed version of Eruption (i.e., interpreted instrument 
readings in terms of probability that the volcano erupts) 
showed a bias in the allocation of funds between the two 
systems in the hazard scenario similar to that shown by 
participants who had never used either version of Eruption ; 
both groups allocated funds more equitably between the 
two systems when the hazard scenario was positively 
framed (system reliability described in terms of percent of 
time system is in working order). This bias toward more 
equitable fund distribution in the positive frame is the 
same bias observed by Quattrone and Tversky (1988) 
among undergraduates at Stanford University and the 
University of California at Berkeley when asked to respond 
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FIGURE 3: Comparison of funds allocated to the first (less reliable) system in the volcanic hazard scenario as a func¬ 
tion of Eruption frame and scenario (question) frame. 
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to a structurally similar decision problem addressing crime 
rates. In both cases the bias may be attributable to the ra¬ 
tio-difference principle that, for problems structurally simi¬ 
lar to the volcanic hazard scenario, tends to reduce the 
perceived difference between the two systems in the posi¬ 
tive frame, encouraging a more equitable allocation of 
funds between the two systems. 

Interestingly, the allocation of funds in the volcanic 
hazard scenario is reversed among those participants who 
used the positively framed graphics-based version of Erup¬ 
tion (i.e., interpreted instrument readings in terms of prob¬ 
ability that the volcano remains dormant); they allocated 
their funds less equitably in the volcanic hazard scenario 
when it was positively framed. For some reason this group 
is relatively risk averse, more strongly preferring the more 
reliable monitoring system (system 2) even when the per¬ 
ceived difference between the two systems is minimized 
using the ratio-difference principle. We speculate that in 
framing both Eruption and the volcanic hazard scenario 
positively, the participant may assess the volcanic hazard 
not in terms of the likelihood of an eruption, but rather in 
terms of the likelihood that the current status will be main¬ 
tained (i.e., that the volcano remains dormant). From this 
perspective, the current status (dormancy) may be viewed 
as an achieved gain that would be lost if an eruption 
occurs. It is well established that people have a greater 
tendency for risk aversion when assessing risk from the 
perspective of losing an achieved gain (Kahneman and 
Tversky, 1984). In contrast, participants switching from a 
positively framed version of Eruption, where the focus is 
on maintaining dormancy, to a negatively framed hazard 
scenario, where the focus is on the likelihood of an erup¬ 
tion, are forced to switch their perspective from that of 
maintaining an achieved gain (dormancy) to that of miti¬ 
gating a loss (an eruption). Just as people have a greater 
tendency for risk aversion on the domain of gains, they 
similarly have a tendency for risk seeking in the domain of 
losses (Kahneman and Tversky, 1984). 

Impact of Data Presentation Format 

Regardless of the psychophysics underlying their deci¬ 
sion making, the impact of the different scalar data presen¬ 
tation formats is clear; students who used the text-based 
data presentation format of Eruption made more skilled 
decisions in the volcanic hazard scenario in the sense that 
their decisions were invariant to framing effects. This indi¬ 
cates that the text-based presentation format has a greater 
impact on the reducing the sensitivity to framing than the 
graphics-based presentation format. 

While both the text-based and graphics-based interfa¬ 
ces were informationally equivalent, the ease of accessing 
this information was significantly greater in the graphics- 
based version in at least two respects. First, when a data 
point is plotted graphically, it's vertical location provides 
an immediate visual clue as to which range of readings the 
point belongs to; low, intermediate, or high (Fig. 2). In con¬ 
trast, users of the text-based interface must compare the 
numeric value from the instrument to the limiting values 
in the low, intermediate, and high ranges for that data type 
in their risk tables. We note that the limiting values 
between the different ranges are different for each type of 
instrument. For example, a seismometer reading of 10 
(earthquakes per day) corresponds to a relatively low risk 


of imminent eruption whereas a geodimeter reading of 10 
(cm) corresponds to a very high risk of imminent eruption. 
Second, as the simulation proceeds, the graphics-based 
presentation format provides easy visual access to previ¬ 
ous instrument readings (Fig. 2). This historical informa¬ 
tion is only available in the text-based version if the 
participant remembers, writes down, or graphs previous 
instrument readings. For these reasons we argue that the 
interpretation of the instrument readings is easier with the 
graphics-based format. Despite its enhanced usability, the 
graphics-based data presentation is less effective in 
improving subsequent decisions. 

The greater effectiveness of the text-based presentation 
format is consistent with previous work demonstrating 
that user interfaces optimized for ease of use and minimal 
cognitive effort are not always effective for learning pur¬ 
poses (Svendsen, 1991; Trudel and Payne, 1995) and is con¬ 
sistent with our previous work demonstrating that a 
computer-based laboratory exercise was more effective 
when students were required to exert more cognitive effort 
by performing the underlying algebraic analyses them¬ 
selves rather than having them done automatically by the 
computer (Renshaw et al., 2000). However, this conclusion 
is inconsistent with our other previous findings demon¬ 
strating that students trained using CAI to interpret three- 
dimensional data using color-coded maps have greater 
understanding of the underlying data than those trained to 
use traditional monochrome contour maps requiring 
greater mental effort to interpret (Taylor et al, 2004). 

Data classification, algebraic manipulations, and the 
interpretation of spatial data are distinct cognitive activ¬ 
ities and thus it is not surprising that enhancing the usabil¬ 
ity of CAI software by enhancing each of these activities 
differentially impacts the exercise's effectiveness in 
improving higher order cognition. From a software design 
perspective, however, it would be helpful to have some 
guidelines, which can be generalized, that apply to all 
these activities and indicate conditions under which soft¬ 
ware usability can be increased without sacrificing its 
impact on higher order skills. 

Taken together, the results from our CAI investigations 
indicate that cognitive effort, by itself, is not a sufficient cri¬ 
terion for determining learning effectiveness. We argue 
that the key consideration in controlling learning effective¬ 
ness is not total cognitive effort, but instead how the effort 
is reduced. Both in Eruption and in our quantitative analy¬ 
sis laboratory, cognitive effort was reduced by eliminating 
essential components of the analysis —the classification of 
the data into appropriate risk categories and the underly¬ 
ing algebraic components. The hope in both cases was that 
the resulting laboratory exercise might help participants 
focus their attention on the core conceptual ideas without 
the distraction of the quantitative details. Instead, we 
found that students need to understand the underlying 
analyses before the core conceptual ideas become clear. 

In contrast, the enhancement of three-dimensional 
maps with color coding, although reducing the cognitive 
effort required to understand the map, does not eliminate 
the essential step of interpreting the data. Instead, the 
color-coded maps assisted participants in completing an 
essential task. Color can be a natural quantifier, is intui¬ 
tively linked to some concepts, and can be easy to learn 
and remember. This is not to say that color always aids 
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conceptual interpretations; it can sometimes lead to misin¬ 
terpretations (Tufte, 1990). We conclude that in designing 
an optimal user interface for computer-assisted learning it 
is possible the simultaneously reduce cognitive effort and 
improve learning effectiveness, but only if the design 
changes assist students in completing essential tasks rather 
than eliminate the tasks by automation. 

This conclusion is consistent with research examining 
the role of working memory in learning. Working memory is 
a hypothesized memory system for temporarily managing 
information used for complex cognitive tasks such as learn¬ 
ing, reasoning, and comprehension (Baddeley, 2003). The 
system has separate processing mechanisms for verbal and 
visuospatial information, the phonological loop and visuo- 
spatial sketchpad, respectively. Assisting participants in 
completing an essential tasks by providing information in 
multiple formats maximizes their use of working memory 
resources, thus appearing to increase the flexibility of a learn¬ 
er's experience and to be particularly effective at freeing up 
cognitive resources for deeper information processing (Pai- 
vio, 1990; Pass et al., 2003; Van Merrienboer et al., 2003). In 
fact, there is evidence that such partitioning aligns directly 
with deeper processing in a fairly intuitive way: integrating 
spatial and verbal information is in and of itself a form of 
deeper processing (Mayer, 1997; Mayer and Moreno, 2002). 
This is particularly true when an information type is particu¬ 
larly well suited to an information format, for example spa¬ 
tial data (Brunye et al., 2008; 2006). In the present study, the 
graphic presentation of the volcanic instruments appealed 
only to the visuospatial processes of working memory. 

CONCLUSIONS 

Among users of the CAI laboratory Eruption, there is a 
significant interaction between data presentation format. 
Eruption framing, and volcanic hazard scenario framing. 
When separated based on presentation format, the interac¬ 
tion only remains for the graphics-based presentation for¬ 
mat, indicating that is has less of an impact on reducing 
sensitivity to framing effects in decision making than the 
text-based data presentation format. The graphical data 
presentation appears to result in a less effective educational 
exercise, perhaps due to the reduction in cognitive effort 
required to understand the instrument readings or by 
engaging only visuospatial working memory processes. 
However, we suggest that cognitive effort, by itself, is not 
a sufficient criterion for determining learning effectiveness. 
Because students need to understand the underlying analy¬ 
ses before the core conceptual ideas become clear, in 
designing optimal user interfaces for CAI software, efforts 
to reduce cognitive effort and improve learning effective¬ 
ness must focus on design changes that assist students in 
completing essential tasks rather than eliminating the tasks 
by automation. 

Acknowledgments 

This research was supported by U.S. National Science 
Foundation Grant No. CMMI-0433203. 

REFERENCES 

Baddeley, A., 2003, Working memory: Looking back and looking 

forward: Nature - Neuroscience Reviews, v. 4, p. 829-839. 


Brown, J.S., Collins, A., and Duguid, P., 1989, Situated cognition and 
the culture of learning: Educational Researcher, v. 18, p. 32-42. 

Brunye, T.T., Taylor, H.A., and Rapp, D.N., 2008, Can you say 
that again: Repetition and integration in procedural multime¬ 
dia learning: Appl. Cognitive Psychology, v. 22, p. 877-895. 

Brunye, T.T., Taylor, H.A., Rapp, D.N., and Spiro, A., 2006, Multi- 
media presentation of procedures: Influences on memory for 
structure and content: Appl. Cognitive Psychology, v. 20, p. 
917-940. 

Butler, K.A., 1996, Usability engineering turns 10: Interactions, v. 
3, p. 59-75. 

Green, T.R.G., and Payne, S.J., 1984, Organization and learnability 
in computer languages: Int. J. Man.-Mach. Stud., v. 21, p. 
300-311. 

Hornbaek, K., 2006, Current practice in measuring usability: 
Challenges to usability studies and research: Int. J. Hum.- 
Comput. Stud., v. 64, p. 79-102. 

Jou, J., Shanteau, J., and Harris, R.J., 1996, An information proc¬ 
essing view of framing effects: The role of causal schemas in 
decision making: Memory & Cognition, v. 24, p. 1-15. 

Kahneman, D., and Tversky, A., 1984, Choices, values, and 
frames: American Psychologist, v. 39, p. 341-350. 

Kirk, R.E., 1982, Experimental design: Procedures for the behav¬ 
ioral sciences: Belmont, CA, Wadsworth. 

Maxwell, S.E., 1990, Designing experiments and analyzing data: 
A model comparison perspective: Belmont, CA, Wadsworth. 

Mayer, R.E., 1997, Multimedia learning: Are we asking the right 
questions?: Educ. Psychologist, v. 32, p. 1-19. 

Mayer, R.E., and Moreno, R., 2002, Aids to computer-based multi- 
media learning: Learning and instruction, v. 12, p. 107-119. 

McCaffery, E.J., Kahneman, D.J., and Spitzer, M.L., 1995, Framing 
the jury: Cognitive perspectives on pain and suffering 
awards: Virginia Law Review, v. 81, p. 1341-1420. 

McNeil, B.J., Pauker, S.G., Sox, H.C., and Tversky, A., 1982, On 
the elicitation of preferences for alternative therapies: New 
England Journal of Medicine, v. 306, p. 1259-1262. 

Nielsen, J., 1994, Heuristic evaluation, in Nielsen, J., and Mack, R.L., 
eds., Usabilityinspection methods: New York, NY, Wiley. 

Ormrod, J.E., 1995, Human learning: Upper Saddle River, NJ, 
Prentice Hall. 

Paivio, A., 1990, Mental representations: New York, NY, Oxford 
University Press. 

Pass, F., Renkl, A., and Sweller, J., 2003, Cognitive load theory 
and instructional design: Recent developments: Educ. Psy¬ 
chologist, v. 38, p. 1-4. 

Quattrone, G.A., and Tversky, A., 1988, Contrasting rational and 
psychological analyses of political choice.: American Political 
Science Review, v. 82, p. 719-736. 

Renshaw, C.E., Johnson, G.D., and Kvale, K., 2000, A laboratory 
exercise to determine dinosaur speeds using dimensional 
analysis: J. Geosci. Educ., v. 48, p. 342-346. 

Renshaw, C.E., Taylor, H.A., and Reynolds, C.H., 1998, Impact of 
computer-assisted instruction in hydrogeology on critical 
thinking skills: J. Geoscience Education, v. 46, p. 274-279. 

Rubens, W., Emans, B., Leinonen, T., Skarmeta, A.G., and Simons, 
R.-J., 2005, Design of web-based collaborative learning envi¬ 
ronments. Translating the pedagogical learning principals to 
human computer interface: Computers & Educ., v. 45, p. 
276-294. 

Salomon, G., 1979, Interaction of media, congnition, and learning: 
An exploration of how symbolic forms cultivate mental skills 
and affect knowledge acquisition: San Francisco, CA, Jossey- 
Bass Inc. 

Sheridan, M.F., Reitan, P., Hodge, D.S., Bursik, M.I., and Fink, 
J.H., 1993, Eruption!: EOS, Trans. Amer. Geophys. Union, v. 
74, p. 82-83. 

Slamecka, N.J., and Graf, P., 1978, The generation effect: Delinea¬ 
tion of a phenomenon: Journal of Experimental Psychology: 
Human Learning and Memory, v. 4, p. 592-604. 


92 Barclay et al. 


J. Geosci. Educ. 59, 85-92 (2011) 


Svendsen, G.B., 1991, The influence of interface style on problem 
solving: Int. J. Man.-Mach. Stud., v. 35, p. 379-397. 

Taylor, H.A., Renshaw, C.E., and Choi, E., 2004, The effect of mul¬ 
tiple formats on understanding complex visual displays: J. 
Geosci. Educ., v. 52, p. 115-121. 

Taylor, H.A., Renshaw, C.E., and Jensen, M.D., 1997, Effects of 
computer-based role-playing on decision making skills: J. 
Educational Computing Research, v. 17, p. 147-164. 

Trudel, C.-I., and Payne, S.J., 1995, Reflection and goal manage¬ 
ment in exploratory learning: Int. J. Hum.-Comput. Stud., v. 
42, p. 307-339. 

Tufte, E.R., 1990, Envisioning information: Cheshire, CT, 
Graphics. 

Tversky, A., and Kahneman, D., 1982, The framing of decisions 
and the psychology of choice, in Hogarth, R., ed.. New direc¬ 


tions for methodology of social and behavioral science: Ques¬ 
tion framing and response consistency: San Francisco, CA, 
Jossey-Bass, p. 3-20. 

Tversky, A., and Kahneman, D.J., 1986, Rational choice and the 
framing of decisions: J. Business, v. 59, p. S251-S278. 

Van Merrienboer, J.J.G., Kirschner, P.A., and Kester, L., 2003, Tak¬ 
ing the load off a learner's mind: Instructional design for 
complex learning: Educ. Psychologist, v. 38, p. 5-13. 

Wills, T.W., Soraci, S.A., Chechile, R.A., and Taylor, H.A., 2000, 
"Aha" effects in the generation of pictures: Memory & Cogni¬ 
tion, v. 28, p. 939-948. 

Winer, B.J., Brown, D.R., and Michels, K.M., 1991, Statistical prin¬ 
ciples in research design: New York, McGraw-Hill. 

Wittrock, M.C., 1989, Generative processes of comprehension: 
Educational Psychologist, v. 24, p. 345-376. 


